Precision Feedback
MPOG Retreat
October 18, 2024
Zach Landis-Lewis, PhD,
MLIS
zachll@umich.edu
Acknowledgements
● Project team members: Nirav Shah, Allison Janda, Allen Flynn, Anjana Deep Renji,
Yidan (Eden) Cao, Hana Chung, Patrick Galante, Peter Boisvert, Mark Dehring,
Kate Buehler, Gan Shi, Andrew Krumm, Kelley Kidwell
● Mentors: Charles Friedman, Anne Sales, Brian Zikmund-Fisher, Sachin Kheterpal
● Many other DLHS and U of M colleagues and former DISPLAY Lab team members
● Funder: NIH National Library of Medicine
○ 1K01LM012528 (PI: Landis-Lewis)
○ 1R01LM013894 (PI: Landis-Lewis)
2
● Developing infrastructure for health care improvement
● Evaluation of information tools to understand impact
● Individual-level, cognitive, clinician-focused interventions
● Information and knowledge-centric lens
3
My perspective: Health informatics
We need better tools for practice-based learning
○ Healthcare professionals must learn continually in many ways
○ We have a wealth of clinical
practice data to support learning
○ However, tools that provide
feedback about clinical practice
are not good at keeping up
Landis-Lewis Z, Cao Y, Chung, H, Boisvert P, Renji AD, et al. Modeling Precision Feedback Knowledge for Healthcare Professional Learning
and Quality Improvement. 2024 AMIA Annu Symp Proc. Accepted 28 June 2024.
I would like to give
you some feedback
Audit and feedback (A&F)
○ Defined as the delivery of a performance summary to healthcare
professionals or teams
○ An implementation strategy that is commonly used and studied
○ 2012 systematic review showed wide variation in effectiveness
■ 140 randomized controlled trials
■ 4.3% median absolute increase in desired practice
■ Interquartile range: 0.5% to 16%
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback:
effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012 Jun 13;2012(6):CD000259.
Audit and feedback (A&F), continued
○ Growing recognition that evidence has not changed for decades
○ Dramatic increase in trials of A&F
■ 25% (75/293) of all A&F trials (since the 1970s) were published between 2016-20
○ 15 best practices established (Brehaut 2016)
○ Formation of the A&F Metalab to build a cumulative science
Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, Ivers N, Grimshaw JM. Practice Feedback Interventions: 15 Suggestions for
Optimizing Effectiveness. Ann Intern Med. 2016 Mar 15;164(6):435-41. doi: 10.7326/M15-2248. Epub 2016 Feb 23. PMID: 26903136.
Grimshaw JM, Ivers N, Linklater S, Foy R, Francis JJ, Gude WT, Hysong SJ. Reinvigorating stagnant science: implementation laboratories and a
meta-laboratory to efficiently advance the science of audit and feedback. BMJ quality & safety. 2019 May 1;28(5):416-23.
Ivers N. Updating the Cochrane Audit & Feedback Review - Completing a decade long odyssey [Internet]. International Audit & Feedback Summit
2022; 2022 Oct 26 [cited 2024 Sep 3]; Virtual Conference. Available from: https://vimeo.com/765774137
A&F research challenges
○ Quality dashboards are everywhere, but
infrequently used
○ Growing interest in studying
how and when
different kinds of feedback are effective
○ Growing interest in understanding
engagement and tailoring of feedback
Keeping complexity in focus
8
People are different
Context matters
Things change
Source: https://www.pchalliance.org/news/how -do-you-change-behavior
9
• Prioritizes coaching and appreciation messages
• Uses estimates of the motivational potential of feedback
messages
• Supports performance improvement and sustainment
Precision Feedback
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Precision feedback: Example 1
10
Dear Dr. Jane,
You reached the top 10% peer benchmark for the measure
PUL-01: Protective Tidal volume, 10mL/Kg PBW.
90%
(26/29)
97%
(29/30)
91%
(29/32)
100%
80%
60%
40%
20%
0%
You Top 10% Peer Benchmark
Sep
‘21
Oct
‘21
Nov
‘21
Dec
‘21
Jan
‘22
Feb
‘22
Mar
‘22
Apr
‘22
May
‘22
Jun
‘22
Jul
‘22
Aug
‘22
90% Goal
11
Hello Dr. Jane,
You are not a top performer for avoiding postoperative nausea
and vomiting (PONV-03):
BENCHMARK
Precision feedback: Example 2
12
Below is your MPOG quality performance report. For a case-
by-case breakdown of any measures’ result, click on the link
at left to visit your quality dashboard.
Hello Dr. Jane,
Congratulations on your high quality of care for the measure
PUL-01: Protective Tidal volume, 10mL/Kg PBW. Your
performance was 97% (29/30), in the top 10% of your peers.
Precision feedback: Example 3
13
Precision
feedback
system
14
Feedback with high
motivational potential
P re c is io n
feedback
system
15
Feedback with high
motivational potential
How and when does
feedback work?
P re c is io n
feedback
system
16
What data and
messages are available?
Feedback with high
motivational potential
How and when does
feedback work?
P re c is io n
feedback
system
17
+
What data and
messages are available?
What does the feedback
recipient prefer?
Feedback with high
motivational potential
How and when does
feedback work?
P re c is io n
feedback
system
18
+
What data and
messages are available?
What does the feedback
recipient prefer?
+
What is optimal in the
recipient’s context?
Feedback with high
motivational potential
How and when does
feedback work?
P re c is io n
feedback
system
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
High
performance
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
High
performance
High performance
and achievement
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
High
performance
High performance
and achievement
Low
performance
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
High
performance
High performance
and achievement
Low
performance
Low performance
and improvement
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
High
performance
High performance
and achievement
Low
performance
Low performance
and improvement
Low performance
and loss
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
“Your performance
is above the goal”
“You reached the
goal”
High
performance
High performance
and achievement
Low
performance
Comparisons
to goals and
standards
“Your performance
is approaching the
goal”
Low performance
and improvement
“Your performance
is below the
standard”
“Your performance
dropped below the
standard”
Low performance
and loss
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
“Your performance
is above the goal”
“You are a top
performer”
“You reached the
goal”
“You reached the
top performer
benchmark”
High
performance
High performance
and achievement
Low
performance
Comparisons
to goals and
standards
Social
comparison
“Your performance
is approaching the
goal”
Low performance
and improvement
“You are not a top
performer”
“Your performance
is below the
standard”
“Your performance
dropped below the
standard”
Low performance
and loss
“Your performance
is approaching the
benchmark”
“Your performance
dropped below
average”
Appreciation feedback
● Identify accomplishments
and achievement
● Motivate performance
sustainment
Evaluation feedback
● “Standard” audit and feedback
● Show current
standing /
performance level
● Compare
performance
● Show change in
performance
Coaching feedback
● Identify learning opportunities
and progress
● Motivate performance
improvement
“Your performance
is above the goal”
“You are a top
performer”
“You reached the
goal”
“You reached the
top performer
benchmark”
“Your performance
is improving”
High
performance
High performance
and achievement
Low
performance
Comparisons
to goals and
standards
Social
comparison
Comparator
not specified
“Congratulations on
your consistently
high performance”
“Your performance
is approaching the
goal”
“You reached a new
high performance
level.”
Low performance
and improvement
“You are not a top
performer”
“You may have an
opportunity to
improve”
“Your performance
is below the
standard”
“Your performance
dropped below the
standard”
“Your performance
has dropped”
Low performance
and loss
“Your performance
is approaching the
benchmark”
“Your performance
dropped below
average”
Motivational potential of a feedback message
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Motivational potential of a feedback message
Motivational
potential
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Motivational
potential
Motivating
information
Motivational potential of a feedback message
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Motivational
potential
Motivating
information
Surprisingness
moderates
Motivational potential of a feedback message
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Motivational
potential
Motivating
information
Preference
Surprisingness
moderates
Motivational potential of a feedback message
Landis-Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Syst. 2024 Apr
9;8(3):e10419. doi: 10.1002/lrh2.10419. PMID: 39036537; PMCID: PMC11257058.
Precision feedback knowledge base
Causal pathways: 10
Message templates: 22
Personas: 8
Vignettes: 10
https://github.com/Display -Lab/knowledge -base
Landis-Lewis Z, Cao Y, Chung, H, Boisvert P, Renji AD, et al. Modeling Precision Feedback Knowledge for Healthcare
Professional Learning and Quality Improvement. 2024 AMIA Annu Symp Proc. Accepted 28 June 2024.
Cluster-randomized trial (May - Oct 2024)
38
●
Comparison:
Precision feedback-enhanced email vs standard feedback email
●
Primary outcome:
Measure success rate (
M
) for operative cases of anesthesia
providers
● We calculate
M
a s fo llo ws :
Numerator: Sum of all operative case measurement
successes
Denominator: Number of all operative case measurements
Cluster-randomized trial, continued
39
●
Hypothesis:
Providers receiving precision feedback will increase
a) measure success rate
b) email engagement (click-through and dashboard login rates)
when compared with providers receiving standard A&F emails
● We will also assess unintended consequences in a mixed-methods process
evaluation
● Opt-out recruitment goals met, currently enrolled ~11,440 providers
● Data collection ending in November 2024
40
Future work
and research
opportunities
“Your team has
sustained high
performance for
_______ over
the last year”
Learning Network
● Analysis of clinical data to identify coaching and appreciation opportunities
● Precision feedback reporting for teams, supporting quality improvement
● Accommodating preferences of feedback recipients, sources, and messengers
● Appreciation feedback as a motivator for performance sustainment
Precision Feedback
MPOG Retreat
October 18, 2024
Zach Landis-Lewis, PhD,
MLIS
zachll@umich.edu
Thank you!
Descriptive models:
Performance metrics
Feedback message
templates
Causal pathway models
Required inputs
Recipient ID
Performance metric ID
Tim e inte rva ls
Performance levels
Recipient
Ind ivid u a l
Team
Comparator
Benchmarks
Go a ls
Optional inputs
Recipient preferences
Precision Feedback
P ip e line
“You are a top
performer for
_________”
Performance
data
Knowledge base
Feedback Recipient
Algo rith m s :
Signal detectors
P rio ritiza tio n
a lg o rith m s
Landis-Lewis Z, Cao Y, Chung, H, Boisvert P, Renji AD, et al. Modeling Precision Feedback Knowledge for
Healthcare Professional Learning and Quality Improvement. 2024 AMIA Annu Symp Proc. Accepted 28 June 2024.
Possible Message
“You reached the top
10% benchmark for
the measure SUS-01”
Motivation
(Regulatory Fit)
Clinical
process
sustainment
Health
outcome
improvement
Intervention Mechanism Proximal outcome Distal outcome
Motivating information
- Positive comparison: Better than comparator
- Trend: Improving
- Achievement: Improving to reach comparator’s level
Surprisingness information
- Size of positive comparison
- Slope/value of positive trend
- Achievement recency
- Message recency
Preconditions Moderators
Causal pathway model: “Social gain”
id
measure
id
time
interval
provider
performance
top 10%
benchmark Goal
1 SUS-01 2023-08 88 94 90
2 SUS-01 2023-09 89 96 90
3 SUS-01 2023-10 88 95 90
4 SUS-01 2023-11 92 94 90
5 SUS-01 2023-12 96 95 90
Positive trend
(88, 92, 96)
Positive comparison
(96 - 95 = 1)
Negative
comparison
(92 - 94 = -2)
Achievement = True
“You reached the Top 10% benchmark for the
measure SUS-01”
Performance
data
Motivating
performance
information
Possible message:
